Computer assisted intramedullary rod surgery system with enhanced features

ABSTRACT

A computer assisted surgery system is described for assisting a surgeon in aligning a drill with the interlocking holes of an implanted intramedullary (IM) rod used for fixation of long bone fractures. With the IM rod inserted, a localizing device measures the pose of a tracked adapter attached to the rod&#39;s exposed end. Approximate AP and lateral fluoroscopic x-ray images are acquired of the end of the rod with the interlocking holes. Image processing algorithms determine the actual position of the rod and calculate an adjustment to the pose of the tracked adapter and IM rod. Using the adjusted pose information, the system displays, in three roughly orthogonal views, a representation of the drill trajectory relative to the images of the IM rod and relative to a graphic representation of the IM rod.

RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. application Ser. No.09/683,107, filed Nov. 19, 2001, titled “Computer AssistedIntramedullary Rod Surgery System With Enhanced Features,” which claimsthe benefit of U.S. Provisional Application No. 60/249,697 filed Nov.17, 2000. The disclosures of the 107 and 697 applications are herebyincorporated in their entirety.

FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] [Not Applicable]

MICROFICHE/COPYRIGHT REFERENCE

[0003] [Not Applicable]

BACKGROUND OF THE INVENTION

[0004] This invention relates to a computer assisted surgery system foruse in inserting interlocking screws in an intramedullary rod.

[0005] A current surgical treatment for fractures of the shaft of longbones (e.g., femur and tibia) is the insertion of an intramedullary rod(IM rod). These devices are relatively rigid devices inserted into oneend of the bone and down the center canal of the bone shaft, such thatthe fracture site is bridged. Transverse holes in either end of the IMrod receive screws inserted transversely through the bone in order tolock the two bone fragments relative to one another. The insertion ofthe screws farthest from the IM rod insertion hole is currently adifficult and time consuming procedure requiring numerous x-ray images.An intraoperative x-ray machine (C-arm) is repeatedly fired andreoriented until it is exactly aligned with the transverse holes asevidenced by x-ray images displaying the holes as “perfect circles”. Toestablish a starting point, the surgeon uses further x-ray images toalign the drill tip with the images of the holes. The surgeon then usesthe source-to-receiver axis of the C-arm as an external reference framealong which the long axis of the drill is oriented. Even after this,several attempts may be required to drill the holes into the bone andthrough the transverse holes.

[0006] Several alternative approaches have been employed in an attemptto speed this process. External jigs have been tried with little successbecause inaccuracies in the jig, inaccuracy of the mounting between jigand IM rod, and deformation of the IM rod accumulate to cause the finaljig hole positions to be unreliably aligned with the IM rod holes.Radiolucent drills and drill guides and laser sighting devices have beendeveloped which, in the best cases, improve the speed and accuracy ofhole placement, but still require a significant number of x-ray imagesto be obtained in order to first achieve a C-arm orientation thatproduces “perfect circles” in the images.

[0007] Image-guided approaches have been developed, but these toorequire the “perfect circle” alignment of the C-arm. Most image guidedsystems display the drill trajectory over “perfect circle” images of theIM rod. One system assists the surgeon in correctly orienting the C-armto obtain “perfect circles”: “Surgical Navigation Based on FluoroscopyClinical Application for Computer-Assisted Distal Locking ofIntramedullary Implants”, Suhm, et.al., Computer Aided Surgery5:391-400, 2000. Another difficulty with existing image guided systemsis that the surgeon must align the drill guide while viewing an “end on”representation of the drill guide, which can be quite challenging.

[0008] Several devices have been described (U.S. Pat. Nos. 5,411,503,5,540,691, 6,074,394, 6,081,741) in which an emitter is inserted intothe IM rod, down to the level of the interlocking holes, and transducerson the drill guide report the position of the drill trajectory relativeto the holes. These devices, however, require equipment dedicated tothis one surgical task, require the extra step of inserting an emitterto the level of the hole, and typically provide only rudimentary “endon” representations of the drill trajectory.

[0009] U.S. Pat. No. 6,285,902, incorporated herein by reference,entitled “Computer Assisted Targeting Device for Use in OrthopaedicSurgery” describes a system in which, preferably, orthopaedic surgicaltools outfitted with infrared LEDs are tracked by an optical localizingdevice. The poses of these tools are determined and graphicrepresentations of the tools are superimposed on standard intraoperativex-ray images. This allows the surgeon to view, in real time, theposition of the tool or tools with respect to an imaged body part oranother tool or tools. In the preferred embodiment, a drill guideoutfitted with infrared LEDs is tracked and the trajectory of its boreis displayed on the x-ray image of the involved bone. This allows asurgeon to accurately predict the trajectory of a guide pin that passesthrough the bore of the drill guide. The guide pin, once inserted, isused as a reference for the insertion of implantable cannulated screws.

[0010] An alternative embodiment of the previous invention, described inthe referenced patent, allows its use in the insertion of distalinterlocking screws in an intramedullary (IM) rod by displaying thedrill guide trajectory relative to a computer generated representationof a cross-section of the IM rod. The current invention is anenhancement to the previous invention that adjusts the graphicrepresentations of the IM rod based on information developed from thex-ray images. This facilitates the more accurate alignment of a drillthrough the holes and eliminates the need to align the x-ray beam withthe holes in the IM rod. This can significantly reduce the amount ofradiation involved in the procedure and reduce the time required toinsert the screws.

BRIEF SUMMARY OF THE INVENTION

[0011] Accordingly, one objective of the present invention is to providea computer assisted surgery system for positioning an instrumentrelative to a portion of a surgical implant. More specifically, itassists a surgeon in drilling a hole through a long bone and throughtransversely oriented holes in an intramedullary rod (IM rod) during afracture fixation procedure regardless of deformation of the IM rod.

[0012] Another objective of the invention is to provide a technique andapparatus for accurately displaying the trajectory of the drill relativeto the holes of the IM rod.

[0013] Still another objective of the invention is to provide atechnique and apparatus for using x-ray images of the IM rod toaccurately determine the locations of the holes.

[0014] These and other objects of the present invention are achieved bythe use of a computer assisted surgery system, including a computer, alocalizing device and a display monitor. The system also includes atracked adapter attached to the IM rod and a drill guide, both of whichhave their poses determined by the localizer. With the IM rod insertedin a long bone, and the tracked adapter attached to the exposed end ofthe IM rod the pose of the adapter and the IM rod are measured by thelocalizing device. Two approximately orthogonal x-ray images are thenobtained of the IM rod in the vicinity of the holes. Image processingtechniques are used to accurately determine the location of the IM rodfrom the x-ray images and an adjusted pose is calculated for the IM rod.A graphic representation of the drill trajectory is displayedsuperimposed over the images of the IM rod and over a graphicrepresentation of the IM rod, in order to assist the surgeon in placingthe drill in the proper position relative to the IM rod holes.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

[0015]FIG. 1 is a perspective view of an intramedullary rod insertedinto a femur, a tracked adapter attached to the intramedullary rod, adrill guide, a partial C-arm, the computer assisted surgery system withlocalizer camera and display screen containing images and graphics.

[0016]FIG. 2 is a perspective view of the intramedullary rod withattached tracked adapter.

[0017]FIG. 3 is an example of a screen display of two images and agenerated graphic with superimposed instrument representations.

[0018]FIG. 4 is a perspective view of an intramedullary rod insertedinto a femur, a tracked adapter attached to the intramedullary rod, adrill guide, a partial C-arm, and a example of a screen display of animage with superimposed graphics.

[0019]FIG. 5 is a diagrammatic illustration of the software operationduring pose adjustment.

DETAILED DESCRIPTION OF THE INVENTION

[0020] An embodiment of the image guided system of U.S. Pat. No.6,285,902 teaches a system for placing distal interlocking screws in anIM rod. As shown in FIG. 1, the system is based on a computer (121) thatreceives input from an x-ray imaging device (110) and a localizingdevice (120), and displays surgical instrument representations (123)over x-ray images (125 and 126) in real time. A tracked adapter (129) isattached to the exposed end of the inserted IM rod (130) such that thepose of the rod can be tracked. A drill guide (128) is also tracked, anda representation of its trajectory (123) is overlaid on x-ray images(125 and 126) of the IM rod (130). Additionally, the system displays, ina separate window (127), the drill guide trajectory (123) relative to agraphical representation (124) of a cross-section of the IM rod (130) atthe level of the transverse interlocking holes (131) by projectingmodels of these instruments onto a picture plane (138). During theprocedure, the surgeon uses the AP image (126) of the IM rod (130) toalign the drill guide (128) in the coronal plane, and the lateral image(125) and the cross sectional graphic (124) to align the drill guide(128) in the axial plane.

[0021] The current invention improves upon the previous invention byproviding greater accuracy in generating the graphic representations(124) of the IM rod (130) relative to the drill guide (128), regardlessof bending of the IM rod (130) or minor errors in attaching the adapter(129) to the IM rod (130). This improvement allows the surgeon to usethese relative graphic representations (124 and 123) alone to exactlyalign the drill guide (128) in the axial plane. While the surgeon stilluses the AP view (126) to align the drill guide (128) in the coronalplane, there is no longer a need to rely on the lateral view (125), thusavoiding the difficulty of positioning a drill guide (128) using an“end-on” representation, during the axial alignment of the trajectory.It also eliminates the need for the surgeon to estimate the requiredanteroposterior position of the drill guide tip based on the distancebetween the IM rod and the femoral shaft cortex and the amount of axialrotation of the IM rod. In the preferred embodiment, the IM rod (130) isinserted in the long bone (133) in the usual manner. While the inventionwill be preferably described for drilling holes in the bone (133) forthe interlocking holes (131) in the end of the IM rod farthest from theexposed end, the system may be alternately used for all interlockingscrews and associated implants. As shown in FIG. 2, a tracking device(129), preferably comprising an adapter (136) to the IM rod (130) and anarray of three or more localizing emitters (137), is attached to theexposed end of the IM rod (130). The exposed end of the IM rod (130) iskeyed to the adapter (136) such that the adapter (136) is attached tothe inserted rod in a unique manner and the relationship is known withinthe limits of error of attachment preferably to within a few degrees anda few millimeters. A coordinate frame, A, is defined preferably at theinterface between the IM rod (130) and the tracking device (129) and isin a known and fixed relationship to the localizing emitters (137).Computer models of the features of the tracking adapter (129) and IM rodrelative to coordinate frame A and graphic representations of featuresof the IM rod (130) relative to coordinate frame A are stored in thecomputer's long term memory. A second coordinate frame, Z, is definedrelative to the localizing emitters (137) of the tracking device (129)and is preferably located on the IM rod (130) halfway between thetransverse holes (131). It is oriented with the z-axis coincident to thelong axis of the IM rod (130) and the x-axis parallel to the bore of thetransverse holes (131). Another coordinate frame, G, is selectedrelative the localizing emitters (137) such that its x-axis and y-axisdefine a picture plane (138) upon which instrument representations maybe projected to form an image for display. The z-axis of coordinateframe G is preferably oriented such that it passes through the centersof both distal transverse holes (131), thus causing representations ofboth distal transverse holes (131) to project to the same location onthe picture plane (138). Alternatively, the picture plane (138) may beselected in any pose that is near-orthogonal (e.g., within 20 degrees)to the long axis of the IM rod without departing from the instantinvention. Further, separate picture planes may be selected for eachtransverse hole through which the system is to assist the surgeon ininserting a screw.

[0022] With the tracking device (129) attached to the IM rod (130) andits pose being read by the localizing device, a graphic representationof the IM rod (130) is projected onto the picture plane (138) defined bycoordinate frame G. Because the picture plane (138) is defined to besubstantially perpendicular to the long axis of the IM rod (130), theimage projected on it will be an “end-view” of the IM rod graphicrepresentation. This end-view image is projected onto the picture plane(138) and, as shown in FIG. 3, is transformed into a graphicrepresentation (124) of the IM rod and displayed in a field (127) of thedisplay screen. The software can display different versions of aninstrument representation for different viewing angles. The end-viewversion of the IM rod representation (124) is a pair of semicirclesrepresenting the cross section of the IM rod (130) with a gap betweenthem representing the transverse holes (131). Virtual lines areadditionally displayed as dashed lines extending from the straightportion of the semicircles to emphasize the orientation of thetransverse screw holes (131). The purpose of this representation (124)is to provide the surgeon with improved information regarding theorientation and location of the transverse holes (131) in the axialplane.

[0023] Returning to FIG. 1, the C-arm (110) acquires x-ray images (125and 126) of the bone (133) that include the transverse holes (131) ofthe inserted IM rod (130). These images (125 and 126) need not beexactly anteroposterior or exactly lateral with respect to the IM rod(130) (i.e., “perfect circles” need not be obtained). The C-arm (110)need be oriented only to within about 30 degrees of exactly AP orlateral, and the misalignment may be either axial or oblique. The poseof the C-arm (110) and the pose of the tracking device (129) and itsrelated coordinate frame A are calculated by the localizing device (120)when the image is acquired. If the x-axis (134) or y-axis (135) ofcoordinate frame A is within, preferably, 30 degrees of thesource-receiver axis of the C-arm (110), then the image is consideredlateral or anteroposterier (AP) respectively.

[0024] If the acquired image is determined to be an AP view then, asshown in FIG. 3, the software will generate an AP version of the graphicrepresentation (145) of the IM rod which is intended to highlight thetransverse holes (131). This graphic representation (145) is definedrelative to coordinate frame A, and is overlaid onto the AP image (126)of the IM rod (130). This AP version of the IM rod representation (145)comprises lines along the sides of the transverse holes' image (141),with dashed virtual lines extending from either side to emphasize theorientation of the holes (141). The drill guide representation (123) isdisplayed relative to the IM rod representation (145) as both areoverlaid on the AP image (126) and improves the surgeon's ability toaccurately align the drill guide (128) with the IM rod transverse holes(131) in the coronal plane.

[0025] If the acquired image is determined to be a lateral view then thesoftware will optionally generate a lateral version of the graphicrepresentation (144) of the transverse holes (131). This graphicrepresentation (144) comprises two circles representing the openings ofthe two holes. It is overlaid on the transverse holes (141) seen in thelateral image (125) to improve the surgeons ability to identify thestarting point for the drill. However, this is of less importance whencompared to the utility of the cross-sectional graphic (127).

[0026] Alternatively, the graphic representation (124, 144 and 145) ofthe IM rod (131) may take other forms including 3-D surface models,bitmaps, or other wireframe models. Any version of the graphicrepresentations (124, 144 and 145), regardless of view orientation, thatprovides the surgeon with sufficient information to orient the drillguide (128) relative to the IM rod (130) in a given plane may be usedwithout departing from the instant invention.

[0027] Additionally, as each image is acquired, adjustment of theposition of the graphic representation (124, 144, and 145) of the IM rod(130) is performed to correct for any deviation due to flexure of the IMrod (131) or inaccuracies in attachment of the tracking device (129) orother conditions leading to inaccuracies in localizing the IM rod (130).Turning to FIG. 4, the poses of the IM rod tracking device (129) and theC-arm (110) are recorded at the time of image acquisition. If the C-armsource-receiver axis (150) is within, preferably, 30 degrees of thex-axis or y-axis of coordinate frame Z then adjustment is to beperformed along the y-axis or x-axis, respectively. While the followingadjustment steps are illustrated in FIG. 4, they are preferablyperformed without being displayed to the user. The adjustment isaccomplished by projecting this adjustment axis (151) of the Zcoordinate frame, onto the acquired image (125) using the conicprojection model, and then analyzing the image data along a specificsegment (156) of this projected line (155). Image processing techniquesknown to those skilled in the art threshold the image data within theimage data segment (156) and find the center (158) of the radio-opaqueIM rod image (140). The difference between this image location (158) andthe projected origin (157) of coordinate frame Z is calculated and thedifference value stored. Alternatively, any image processing techniques,or other means for directly measuring the positional error of the IM rod(130) at or near the transverse holes (131) may be used withoutdeparting from the instant invention.

[0028] When the difference value for AP, lateral, or both images havebeen processed, the corresponding Z frame x-axis and y-axis componentsare calculated by techniques known in the art. These differencecomponents are then used to develop an adjustment transformation.Returning to FIG. 3, this adjustment transformation is applied tocoordinate frame A, causing it to rotate such that the IM rod's AP andlateral graphic representations (144 and 145) defined relative to A,will align with the IM rod's x-ray images (140). After the adjustmentrotations, the positions of the cross sectional representation (124),the AP representation (145), and the optional lateral representation(144), which are displayed to the user, more accurately represent theactual position of the IM rod (130).

[0029] Alternately, the overlay of the graphic representations (124, 144and 145) may be corrected by the translation of coordinate frame Ainstead of by rotation. Or, instead, the graphic representations (124,144 and 145) could be altered to effect the correction. For example, ifthe total difference is attributed to bending of the rod, the coordinateframe A could be left unchanged and the graphic representations (123,144 and 145) could be altered to simulate the flexure of the IM rod(130). Any correction means that uses the difference between theexpected and actual positions of the IM rod (130) to modify its graphicrepresentation in such a way to make it more accurate may be usedwithout departing from the instant invention.

[0030] In summary, the software steps required are shown in FIG. 5. Theimage is acquired (160) and the poses of the C-arm and tracking clampdetermined (161). If the y-axis of the Z coordinate frame isnear-aligned with the C-arm (162) then project the Z frame x-axis ontothe image (163), isolate an appropriate segment of image data along theprojected line (164), apply a thresholding algorithm to emphasize the IMrod (165), find the center of the IM rod in the image data (166), andcalculate the difference between the center of the IM rod's image andthe projected Z-frame's origin as modeled (167). If the x-axis of the Zframe is near aligned with the C-arm (168) then project the y-axis ontothe image (169) and perform the image processing steps above (164-167).Calculate an adjustment transformation based on available x-axis andy-axis difference values (170).

[0031] Returning to FIG. 1, once the corrected IM rod representations(124 and 144 and 145 seen best in FIG. 3) are being displayed, thesurgeon prepares to drill the holes in the bone (131). The drill guide(128) generates a trajectory (123) that overlays both image fields (125and 126) and projects onto the graphics field (127) as well. As shown inFIG. 3, the surgeon aligns the drill trajectory (123) to pass throughthe hole markers of the AP IM rod representation (145) and through thehole path on the cross sectional IM rod representation (124) in thegraphics field (127). Once the proper alignment is achieved, the drillis advanced in the drill guide (128). The second hole is prepared in asimilar fashion and confirmatory x-rays are obtained prior to insertingappropriately sized screws. Alternately, self-tapping screws may bedirectly inserted with a suitable instrument.

[0032] While the above description relates to the placement ofinterlocking screws in intramedullary rods placed in long bones, personsskilled in the art will recognize the applicability of this invention toother devices in other locations of the body such as the insertion ofscrews into other implantable devices. Any procedure wherein a firstdevice is positioned relative to a second device at a position that isnot known with sufficient accuracy by use of a localizing device can beperformed in a more accurate manner with this invention.

1. A computer assisted surgery system for positioning a surgicalinstrument relative to a surgical implant within a patient's body, saidsystem comprising: a surgical instrument defining a trajectory; asurgical implant with one or more holes; a localizing device formeasuring the pose of the surgical instrument and the pose of thesurgical implant; means for acquiring x-ray images of the surgicalimplant; means for calculating an adjusted pose for the surgicalimplant, said means for calculating the adjusted pose based oninformation developed from the x-ray images; and means for displaying agraphic representation of the trajectory relative to a graphicrepresentation of the surgical implant, based on the measured pose ofthe surgical instrument and the adjusted pose of the surgical implant;wherein the adjusted pose of the implant compensates for deformation ofthe implant or inaccuracies in localization, so as to assist a surgeonin aligning the trajectory of the surgical instrument with the holes inthe surgical implant.
 2. The computer assisted surgery system of claim 2wherein the information developed from the x-ray images is an adjustmentto be applied to the pose of the implant, said adjustment developedthrough image processing techniques applied to the x-ray images of theimplant.
 3. A method for positioning a surgical instrument relative to asurgical implant, the method comprising: (a) generating a computer modelof the surgical implant; (b) generating a computer model of a trackedadapter; (c) attaching the tracked adapter to the surgical implant; (d)measuring the pose of the tracked adapter with a localizing device; (e)calculating the pose of the implant from the measured pose of thetracked adapter, the computer model of the tracked adapter, and thecomputer model of the implant; (f) calculating a first position in spaceof a reference point on the implant based on the model of the implantand the calculated pose of the implant; (g) acquiring two approximatelyorthogonal 2-D images of the implant; (h) calculating a second positionin space of the reference point on the implant based on its position inthe images; (i) adjusting the calculated pose of the implant based onthe difference between the first and second calculated positions of thereference point on the implant; (j) measuring the pose of the surgicalinstrument with the localizing device; and (k) displaying a graphicrepresentation of the surgical instrument relative to a graphicrepresentation of the implant based on the measured pose of the surgicalinstrument and the adjusted pose of the implant.
 4. The method of claim3 wherein adjusting the calculated pose of the implant comprisesadjusting the graphic representation of the implant.
 5. The method ofclaim 3 wherein calculating a second position of the reference pointcomprises the processing of the images to extract the locations of edgesof the implant.
 6. The method of claim 3 wherein displaying a graphicrepresentation of the surgical instrument relative to the graphicrepresentation of the implant includes the projection of representationson a picture plane approximately orthogonal to the two 2-D images. 7.The method of claim 3 further comprising the step of displaying thegraphic representation of the surgical instrument superimposed on the2-D images.
 8. A method for accurately drilling holes into a long boneso that the holes are aligned with the interlocking holes of anintramedullary rod, the method comprising the steps of: (a) insertingthe intramedullary rod into the bone so as to leave one end of the rodexposed, the opposite end having one or more transverse holes disposedtherein, the rod defining a long axis; (b) attaching a tracked adapterto the exposed end of the rod, the pose of said tracked adaptermeasurable by a localizing device; (c) defining a coordinate frame Grelative to the tracked adapter such that its x-axis and y-axis define apicture plane upon which graphic representations of the intramedullaryrod and surgical instruments may be projected for display to a surgeon;(d) projecting the graphic representation of the intramedullary rod ontothe picture plane defined by coordinate frame G such that an end view ofthe graphic representation of the intramedullary rod is displayed; (e)acquiring one or more x-ray images of the long bone including thetransverse holes of the intramedullary rod using a C-arm; (f) measuringthe pose of the tracked adapter with the localizing device; (g)positioning a drill guide near the bone in the vicinity of theintramedullary rod's transverse holes, said drill guide's pose beingread by the localizing device; (h) projecting a graphic representationof the drill guide on the picture plane defined by coordinate frame G,such that the graphic representation of the drill guide is displayedrelative to the graphic representation of the intramedullary rod; and(i) projecting the graphic representation of the drill guide on one ormore x-ray images or similarly oriented picture planes such that thegraphic representation of the drill guide is displayed relative to anx-ray image or graphic representation of the intramedullary rod.
 9. Themethod for accurately drilling holes into a long bone of claim 8 furthercomprising the steps of: defining a coordinate frame A relative to thetracked adapter; defining a coordinate frame Z relative to coordinateframe A, the z-axis of the Z coordinate frame being aligned with thelong axis of the intramedullary rod, and the origin of coordinate frameZ located at a reference point on the intramedullary rod in the vicinityof the one or more transverse holes; calculating and recording the poseof coordinate frame A from the measured pose of the tracking adapter;and adjusting the recorded pose of coordinate frame A such that theorigin of coordinate frame Z coincides with the reference point in thex-ray images of the intramedullary rod.
 10. The method for accuratelydrilling holes into a long bone of claim 9 wherein the step of adjustingthe recorded pose of coordinate frame A comprises: selecting an x-rayimage of the intramedullary rod containing the reference point;selecting an axis of the Z coordinate frame that is roughlyperpendicular to the long axis of the intramedullary rod and parallel tothe plane of the x-ray image; projecting said axis of the Z coordinateframe onto the image of the rod; identifying a segment of the image dataalong the projected line; applying an image processing algorithm to findthe reference point in the segment of image data; calculating thedifference between the reference point and the projected origin of the Zframe; and calculating an adjustment transformation based on thedifference value.
 11. The method of claim 10 wherein the reference pointis located at the center of the radio-opaque shadow of theintramedullary rod in the segment of image data.
 12. The method foraccurately drilling holes into a long bone of claim 8 wherein thegraphic representation of the drill guide comprises a trajectory line.13. The method of accurately drilling holes into a long bone of claim 8wherein the intramedullary rod comprises two or more transverse holesand the step of defining a third coordinate frame G further comprisesthe steps of positioning the z-axis of the G coordinate frame such thatit passes through the centers of the two or more transverse holes toproject the holes on the same location on the picture plane.